Computation in Recurrent Neural Networks: From Counters to Iterated Function Systems

In the paper we address the problem of computation in recurrent neural networks (RNN). In the rst part we provide a formal analysis of the dynamical behavior of a RNN with a single self{recurrent unit in the hidden layer, show how such a RNN may be designed to perform an (unrestricted) counting task and describe a generalization of the counter network that performs binary stack operations. In the second part of the paper we focus on the analysis of RNNs. We show how a layered RNN can be mapped to a corresponding iterated function system (IFS) and formulate conditions under which the behavior of the IFS and therefore the behavior of the corresponding RNN can be characterized as the performance of stack operations. This result enables us to analyze any layered RNN in terms of classical computation and, hence, improves our understanding of computation within a broad class of RNNs. Moreover, we show how to use this knowledge as a design principle for RNNs which implement computational tasks that require stack operations. This principle is exempliied by presenting the design of particular RNNs for the recognition of words within the class of Dyck languages.